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Executive Summary 


Background 

Allegheny County Department of Human Services (DHS) is using predictive risk modeling (PRM) 
to assist child welfare staff decide which General Protective Services (GPS) referrals to 
investigate and which to screen out. 

A contracted research team developed the Allegheny Family Screening Tool by conducting an 
extensive series of analyses using data from the DHS data warehouse and the child welfare case 
management system to identify factors that are predictive of a child's re-referral to child 
welfare or placement into foster care. The research team developed an algorithm that applies 
weights to a series of factors to assist in determining when a GPS referral should be assigned 
for investigation or screened out. 

Hornby Zeller Associates, Inc. conducted a process evaluation involving stakeholder interviews, 
surveys, and document review to describe Allegheny County's experience, including perceived 
barriers and facilitators, with implementing PRM. 

Methodology 

The timeline for the process evaluation is summarized in Table E-1, which includes a description 
of the strategies employed and the sources used to collect data. 


Table E-1. Schedule of Process Evaluation Methodology 

Pre-Implementation 

Post-Implementation 

Summer 2016 

Fall 2016 

Winter 2016 

Spring 2017 

Interviews with DHS call screening and other 
DHS staff 

Surveys of call 

screeners 

Interviews with DHS 

research and 
practice staff 

Follow-up surveys 
of call screeners 



Interviews with 

external 

stakeholders 



Findings 

Community stakeholders had positive feedback about the presentations introducing 
Predictive Risk Modeling and hope for continued transparency as the County continues to 
implement the Allegheny Family Screening Tool. 

Considerable effort went into informing internal and external stakeholders through community 
meetings about the County's decision to implement PRM. External stakeholders who attended 
the presentations generally found them to be "encouraging" and "informative." They noted the 











County and its team of experts know what they are doing and inviting stakeholders to the 
presentation showed DHS intends to be transparent in its implementation of the tool. 

Stakeholders noted the need to continue to inform community stakeholders about PRM 
progress, outcomes and plans for ongoing maintenance and sustainability. For instance, one 
provider wanted to know what the "disaster plan" is for the tool, as well as what safeguards are 
in place to ensure that transparency will continue in the future, regardless of who is overseeing 
the project. 

Following implementation, stakeholders continue to have a positive reaction to implementing 
the Allegheny Family Screening Tool. Their hope is that the tool will result in increased safety of 
children and enable the County to be more proactive and less reactive in its case practice. 

The Predictive Risk Modeling Tool is facilitating data-driven decision-making with Allegheny 
County staff, but there is further room for system-level change. 

Administrators agreed that the tool will help staff to make an informed decision. During the 
planning period, administrators were confident the tool would lead to more accurate decision¬ 
making. More than half of the call screeners (61%) said they believe the tool is facilitating a 
shift in the workplace environment to be more data-driven. 

Call screening staff report having a good understanding of the Allegheny Family Screening 
Tool, but are mixed on how confident they are in the resulting scores. 

The majority of call screeners voiced some concern about the reliability of the score, with 72% 
stating they thought a score seemed inaccurate occasionally and an additional 11% noting it 
was inaccurate a moderate amount of the time. Half (50%) of the call screeners surveyed said 
they are confident in the tool's ability to accurately assess risk. Full-time call screeners were 
slightly less likely to express confidence than part-time call screeners. The lack of confidence 
stemmed from the concern that the tool is unable to take a family's individual circumstances 
into account; for instance, a family may be receiving services that are improving the family's 
situation. More than half of the call screeners (61%) said they are confident in the research that 
went into developing the tool. 

Call screen staff generally find the Allegheny Family Screening Tool easy to use, and offered 
technical suggestions for improving the Tool's user experience. 

The majority of call screeners understand how the score works and all surveyed said they are 
"adequately prepared to use the tool." Nearly two-thirds of the call screeners (60%) said the 
tool is "easy" or "very easy" to use. Approximately one-third (38%) had no opinion on the visual 
display of the score on the thermometer. More than two-thirds of those with an opinion (70%) 
said the thermometer is helpful to a limited degree or not at all. 



Suggestions offered by call screeners to improve the Allegheny Family Screening Tool were 
primarily related to technical issues. One of the staff suggested the "score needs to be more 
visible." Several screeners remarked that the system is slow, noting it takes a long time for the 
score to generate and the tool sometimes times out. 

Recommendations 

Maintain transparent communication with internal and external stakeholders. 

Stakeholders overwhelmingly applauded the efforts that DHS has made to be transparent and 
to keep them informed throughout the implementation process. It will be important that this 
transparency continue. 

Increase user buy-in. 

Less than half of the call screeners currently view predictive modeling as benefiting the 
screening practice, though more than 60% agreed that the tool is creating a data-driven culture 
within the workplace. An opportunity exists to increase user buy-in. 

Continue to resolve technical issues as they arise, documenting solutions. 

As changes and enhancements are made to the tool, they should be documented to inform 
further tool development, increasing the return on the technological investment. 

Develop benchmarks for implementing predictive risk modeling. 

Benchmarks can be developed to 1) foster buy-in/increase use of the tool for decision-making 
by call screening staff and 2) promote transparency with stakeholders. 
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Introduction 


The Predictive Risk Tool: Development and Goals 

Development of the Allegheny Family Screening Tool 

Allegheny County, Pennsylvania, has a rich source of data to inform its decision making. The 
Data Warehouse^ of the Department of Human Services (DHS) stores data from a wide array of 
sources including, among others, the juvenile and adult correction systems, public welfare and 
behavioral health agencies and programs. Data from the warehouse are available to aid child 
welfare caseworkers and their supervisors, including call screeners, in their decision making. 
Data integration has paved the way for the use of administrative data in predictive risk analytic 
models to target services to children and families most in need. 

Building on a concept first developed in New Zealand^ to target social services to families at 
high risk of using multiple service systems for lengthy periods of time, Allegheny County DHS 
elected to use Predictive Risk Modeling (PRM)^ to help prioritize cases and target services to 
children most at risk. Allegheny County chose to implement PRM specifically to assist child 
welfare call screening staff to decide which General Protective Services (GPS) referrals warrant 
investigation and which should be screened out.^ 

Through a competitive Request for Proposals^ DHS contracted with an international team of 
researchers, led by Rhema Vaithianathan from the Auckland University of Technology in New 
Zealand and joined by Emily Putnam-Hornstein from the University of Southern California, Irene 
de Haan from the University of Auckland, Marianne Bitler from the University of California- 
Irvine, and Tim Maloney and Nan Jiang from the Auckland University of Technology, to develop 
the Allegheny Family Screening Tool. The research team conducted an extensive series of 
analyses using data from the DHS data warehouse and from the County's child welfare case 
management information system, the Key Information and Demographic System (KIDS). The 
analyses identified factors that are predictive of re-referral to child welfare or placement into 
foster care, and produced an algorithm that applies weights to those factors to assist in 
identifying which GPS referrals are more or less at risk of these outcomes. 


^ Allegheny County Analytics 2017 DHS Warehouse http://www.alleghenycountyanalytics.us/index.php/dhs-data- 
warehouse/, accessed May 19, 2017. 

^ Vaithianathan, Rhema, Tim Maloney, Nan Jiang, Irene De Haan, Claire Dale, Emily Putnam-Hornstein, and Tim 
Dare 2012 Vulnerable Children: Can Administrative Data be Used to Identify Children at Risk of Adverse Outcomes? 
Centre for Applied Research in Economics, Department of Economics, University of Auckland. 

^ Allegheny County Department of Human Services 2017 Predictive Risk Modeling in Child Welfare in Allegheny 
County: The Allegheny Family Screening Tool http://www.alleghenycounty.us/Human-Services/News- 
Events/Accomplishments/Alleghenv-Family-Screening-Tool.aspx, accessed May 19, 2017. 

Vaithianathan, Rhema, Emily Putnam-Hornstein, Nan Jiang, Parma Nand, and Tim Maloney 2017 Developing 
Predictive Models to Support Child Maltreatment Hotline Screening Decisions: Allegheny County Methodology and 
Implementation. Centre for Social Data Analytics, Auckland University of Technology. 

^ Decision Support Tools and Predictive Analytics in Human Services RFP http://county.allegheny.pa.us/Human- 
Services/Resources/Doing-Business/Solicitations/Archive.aspx 
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Introduction to the Process Evaluation 


As part of Allegheny County's effort to document and evaluate the implementation of 
predictive risk modeling, Hornby Zeller Associates, Inc. (HZA), a management consulting firm 
specializing in evaluations of public human service programs, was contracted through a 
competitive bid process to conduct the process evaluation of Allegheny County's 
implementation of PRM.® Casey Family Programs and the Human Service Integration Fund of 
The Pittsburgh Foundation provided funding for this evaluation (and a separate impact 
evaluation conducted by Stanford University). This report provides an overall summation of the 
process evaluation conducted by HZA between mid-2016 and early 2017. The report considers 
the steps the County took prior to and during the initial implementation, the reactions of 
internal and external stakeholders to predictive risk modeling in Allegheny, challenges that 
arose and were addressed, as well as lessons learned during the implementation process. 


® Evaluation of a Predictive Risk Modeling Tool for Improving the Decisions of Child Welfare Workers RFP 
http://county.allegheny.pa.us/Human-Services/Resources/Doing-Business/Solicitations/Archive.aspx 
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Methodology 


Data Collection 

Interviews 

Prior to implementation of the Allegheny Family Screening Tool, HZA conducted interviews with 
DHS administrators and staff from the call screening unit to learn about: a) their involvement in 
the implementation of the tool, b) steps taken to prepare call screening staff to use predictive 
risk modeling to inform their decision-making, and c) the call screening process as it existed 
prior to implementation of the tool. In all, 23 staff were interviewed at baseline, including: 3 
administrators, 8 call screen supervisors, and 12 call screeners. Both part- and full-time call 
screening staff were interviewed, with 55% of the call screening staff providing input. These 
interviews were conducted in July of 2016, just prior to the tool being implemented on August 
1, 2016. 

Four months following implementation of the tool, HZA conducted interviews with 
stakeholders internal and external to DHS. Interviews with community partners focused on 
their awareness of the Department's efforts to implement PRM, their hopes for what the tool 
would accomplish and the successes and challenges they expected the County to face. Internal 
stakeholders were asked about their involvement in implementing the tool, the training they 
received and how the Allegheny Family Screening Tool informs or impacts their work. In all, a 
dozen individuals were interviewed post-implementation, half of whom were from the 
Department's Office of Children, Youth and Families (CYF). Other DHS stakeholders included an 
administrator and staff from the Office of Data Analysis, Research and Evaluation. 
Representatives from community service providers, advocacy groups, foundations, and a family 
court judge made up the group of external stakeholders who were interviewed. 

Surveys 

In September 2016, approximately 2 months post-implementation, HZA administered a web- 
based survey to call screeners. A total of 16 of 21 call screeners completed the survey for a 
response rate of 76%. Three-quarters or 12 of the respondents were full-time call screeners. 
More than half of the survey respondents (56%) had worked as call screeners for at least three 
years. 

Using a series of Yes/No and Likert scale questions, call screeners were asked about the training 
they received, the functionality of the tool, visualization of the scores, and the impact of the 
tool on their decision making. Several open-ended questions were also asked to gather input on 
what could be done to improve the use of the tool and the training provided to prepare staff to 
use it. 

Following a meeting with project staff in early February 2017, which included staff from DHS, 
representatives from the research team and the process evaluation and outcome evaluation 
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teams, a decision was made to administer a follow-up survey to call screeners to account for 
improvements that had been made to the tool. 

A total of 18 call screeners responded to the follow-up survey for a response rate of 86%. All 
full-time screeners responded to the survey while 60% of the part-time screeners participated 
in the second survey. Just under half (48%) of the second survey respondents reported working 
as call screeners for at least three years. 

Data Analysis 

Quantitative Analysis 

Quantitative analyses included summary statistics, frequency counts and percentages. 
Aggregate results of the surveys are provided in the Appendices. 

Qualitative Analysis 

Data collected during the stakeholder interviews and through open-ended questions on the call 
screener surveys were carefully reviewed to identify common themes and items of importance. 
The results of the qualitative analysis describe the implementation process from the 
perspective of the stakeholders, a grounded theory approach.^ The qualitative results are also 
used to support and/or explain the quantitative results, where appropriate. 


Charmaz, Kathy 2000 Grounded Theory: Objectivist and Constructivist Methods. In The Handbook of Qualitative 
Research, edited by N. K. Denzin and Y. Lincoln, pp. 509-535. Sage Publications, Inc., Thousand Oaks, California. 
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Findings 


Pre-Implementation: Preparing for Change 

Call Screening Practice 

The primary role of call screeners prior to implementation of the Allegheny Family Screening 
Tool was to gather the information to inform supervisor decision making. Specifically, call 
screeners collected data about the alleged victim(s), perpetrator(s) and the allegations of 
suspected maltreatment. Information was collected from four primary sources: 1) the caller, 2) 
KIDS, 3) the data warehouse and 4) public databases that contain court and jail information. 

This information was provided to supervisors, who made the decision to screen the call in or 
out. The information gathered by call screeners was also provided to caseworkers to aid the 
assessment process after a call was screened in. 

Call screeners reported that it is much easier to collect information from a mandatory reporter 
than from other callers because mandatory reporters are aware of the information they need 
to supply. Regardless, screeners said they collect as much information about the alleged 
victim(s), the child(ren)'s family and the alleged perpetrator (e.g., names, addresses, ages, 
relationships) from the caller as possible, as well as descriptions of the alleged maltreatment. 
Screeners reported that they check KIDS for every referral to determine if there is already a 
case open on the child or family, in which case they provide the information to the responsible 
caseworker, or if the family had past involvement with the Department. 

Beyond the information collected from the reporter and KIDS, cross-sector administrative data 
are available from other County agencies and community providers through a tool commonly 
known as ClientView.^ External databases, such as Prothonotary (the Allegheny County Court 
screen) and PAC file (the Juvenile Court data system), among others, are also searched. When 
the call screeners were asked during the pre-implementation interviews how frequently they 
search the data warehouse for data about the family, some stated they check it for nearly every 
report, while others report that they use it less than half of the time. Staff did report consistent 
use of the court information. 

While a number of call screeners reported that the data they obtain through ClientView contain 
information that is not accurate or up-to-date, such as a previous address for a family that has 
moved, call screeners also reported using ClientView to "fill in gaps" in client information and to 
gain a better picture of a child or family's situation. Call screeners reported taking 10 to 15 
minutes to complete a search in ClientView. Several call screeners, however, reported taking as 
much as an hour, or even several hours for cases with prior history with the Department. When 
asked how long it typically takes to collect information on a referral, including gathering the 
information and completing the call report for the supervisor, staff noted it took between 25 


® Vaithianathan, Putnam-Hornstein, Jiang, Nand and Maloney, 2017, Developing Predictive Risk Models to Support 
Child Maltreatment Hotline Screening Decisions, Auckland University of Technology in New Zealand 
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and 35 minutes on average. A major factor in the time it takes to complete the intake process, 
as described by the call screeners, involves gathering information from the caller; the more 
information the caller has and is able to provide, the longer it takes to complete the screening. 

Sharing Information with Supervisors and Caseworkers 

In addition to describing their data collection processes prior to implementation, call screeners 
were asked how they document and share the information they collect from the various data 
sources with their supervisors. All screening staff noted that they discussed the reports with 
their supervisor and often gave their input, but the ultimate decision to refer the report for 
investigation was made by the supervisor. Thus, call screeners were primarily tasked with 
gathering the information that supervisors used to determine whether to screen calls in for an 
investigation or to screen calls out without further child welfare involvement (referrals may be 
made to appropriate resources). All staff stated that there are sections within the call report 
template that facilitate how the information was documented. For example, there was a 
section labeled "Legal" where information regarding a family's court involvement, if applicable, 
was recorded. According to the call screeners, much of the information collected in the 
screening process went in the "Additional Information" section. The information collected from 
KIDS and through ClientView during the call screening process was also available to 
investigation caseworkers for planning and conducting investigations. 

Activities and Responses to Preparing for Change 

Policy and Practice Changes 

A shift in policy was made to guide several changes Allegheny County introduced to the call 
screening practice in conjunction with PRM. These changes are illustrated in Table 1. Call 
screeners, rather than just collecting information, were now being asked to complete a risk and 
safety rating and to generate a family screening score using the Allegheny Family Screening 
Tool for each child associated with an allegation of maltreatment. Additionally, call screeners 
now make the recommendation to screen non-mandatory GPS reports in or out while 
supervisors became responsible for reviewing and approving the call screeners' 
recommendations. 


Table 1. Changes in Call Screening Policy and Practice 

Before Implementing PRM 

After Implementing PRM 

Screening decisions made based on clinical 
judgment. 

Screening decisions made based on systematic 
analysis of data and clinical judgment. 

Call screeners collect information. 

Call screeners collect information, complete a risk 
and safety rating and initiate generation of a 
family screening score. 

Supervisors make decision to screen calls in or 

out. 

Call screeners make recommendations to screen 
calls in or out and supervisor approves or changes. 
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Informing External Stakeholders 


Considerable effort went into informing external stakeholders about the County's decision to 
implement predictive risk modeling into its call screening process. Community meetings were 
held to introduce the project to external stakeholders, including advocacy groups, service 
providers, court staff and consumer groups. These presentations highlighted the tool's design 
and underlying research, as well as how it would be integrated into call screening decisions. 

The external stakeholders who attended generally found the presentations to be "encouraging" 
and "informative." The degree to which the presentations enhanced their understanding of 
Allegheny's application of predictive risk modeling, however, varied. For instance, one 
community provider found the information to be very helpful and condensed into pieces that 
were easy to comprehend. Another noted she had to attend a few of the presentations to 
understand predictive analytics, because the topic is "very complex." 

The community meetings DHS held for community stakeholders discussed ethical issues the 
County was facing related to implementation of the screening tool. One topic of interest was 
security and privacy, and whether or not the tool would collect or share any new data regarding 
families. Presenters explained that the tool only leverages data that are already collected and 
owned by the County. Other than to use the data in a more structured and consistent manner 
in making a decision to screen in or out a GPS referral, the data are not intended to be used 
other than they have in the past. 

Discussions with stakeholders also invoked the possibility of the screening tool maintaining or 
exacerbating racial or socioeconomic disparities. Allegheny County's historic data suggest that 
racial disparities already exist at many outcome and decision points throughout the child 
welfare system.^ Presenters suggested that ideally, the tool increases transparency and 
consistency in decision-making, as well as reduces the possibility of call screeners needing to 
draw from their own implicit biases. In the spring of 2016, an analysis of the ethical questions 
surrounding the tool was conducted to explore race's possible role in the tool. Ultimately, in 
conjunction with the researchers' finding that including race in the model did not significantly 
improve its accuracy, administrators, in conjunction with ethics and legal staff, determined that 
race would be omitted as a factor for determining the risk score. 

Anticipated Benefits and Chaiienges 

The goal of implementing predictive risk modeling in Allegheny County, according to DHS 
administrators, was broadly to improve decision making. Collectively, administrators listed six 


® Rauktis, Mary E. and Julie McCrae 2010 The Role of Race in Child Welfare System Involvement in Allegheny 
County. Allegheny County Department of Human Services, Pittsburgh, Pennsylvania. 

Dare, Tim, and Eileen Gambrill 2017 Ethical Analysis: Predictive Risk Models at Call Screening for Allegheny 
County. In Vaithianathan, Rhema, Emily Putnam-Hornstein, Nan Jiang, Parma Nand, and Tim Maloney 2017 
Developing Predictive Models to Support Child Maltreatment Hotline Screening Decisions: Allegheny County 
Methodology and Implementation. Centre for Social Data Analytics, Auckland University of Technology. 
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goals related to determinations made at intake, a "key decision point," as one administrator 
expressed it (See Table 2). 


Table 2. Goals of Implementing Predictive Risk Modeling in Allegheny County 

1. 

Change the agency culture to data and research based decision making. 

2. 

Make better and more efficient use of resources, specifically data resources. 

3. 

Make decisions based on as much information as possible. 

4. 

Increase the number of people making call-screening decisions 

5. 

Create a more uniform screening practice. 

6. 

Increase the accuracy of screening decisions. 


Along with the specific goals, administrators expressed several benefits they hoped would 
result from using predictive risk modeling. The most frequent response regarding the intended 
benefits centered on the accuracy of decision making. 

Concerns were also expressed by DHS staff prior to implementation that the volume of 
investigations might increase and that implementation was being done at a time when there 
had been many recent legislative changes, which might complicate the implementation. One of 
the DHS administrators interviewed prior to implementation voiced a concern that some calls 
would be mandatorily screened in based on the resulting risk score, even though the 
information collected from the caller was not suggestive that the report be assessed. DHS 
elected to designate referrals with a score of 18 or higher on the placement risk model as 
mandatory to be screened in for an investigation, although supervisors are able to override the 
mandate if other factors warrant that decision. 

Preparing Call Screeners and Supervisors 

Staff Training 

Call screening staff received training in how to use the tool in their decision making. This 
comprehensive training taught call screeners how to use the tool to generate and then 
interpret the Family Screening Score. It also provided them with an understanding of how 
predictive risk modeling works and what the Allegheny Family Screening Tool intends to 
accomplish. The process which call screeners were to follow upon implementation of PRM was 
described step-by-step with a workflow presented to illustrate the process. A number of case 
scenarios were also used to demonstrate use of the tool in knowing when to mine for 
additional data and apply the results in decision making. 

The training also reviewed changes made to the KIDS interface in response to PRM and the 
added responsibility for call screeners to complete a risk and safety assessment. For instance, 
the training covered where screeners would record the initial risk and safety decision in the 
case management system, along with the factors considered in determining the appropriate 
level of risk and safety. It also included a demonstration of how client service data would be 
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automatically imported into KIDS from the data warehouse, to help the screener document and 
justify the recommendation to screen in or out the GPS report. 

Staff Perspectives on Training 

The survey administered to call screeners two months following the implementation of PRM 
asked if they had received training prior to using the Allegheny Family Screening Tool. All 16 
respondents stated that they had completed the training, with the majority of them stating that 
the training prepared them to use the tool either "somewhat" or "very well," as shown in 
Figure 1. 


50% 

40% 

30% 

20 % 

10 % 

0 % 


Figure 1. How Well Training Prepared Screeners 
to Use the Allegheny Family Screening Tool 

44% 



Not at All Limitedly No Opinion Somewhat Very Well 


When asked what aspect of the training was most helpful, as shown in Figure 2, over a third of 
the call screeners (36%) said that information about how predictive analytics was to be applied 
in Allegheny County 

was most helpful Figure 2. Training Components Found to be Most Helpful 

Use of case 
scenarios to review 
decision-making 
were rated as the 
second most helpful 
part of the training 
with just under a 
third (29%) of the 
call screeners 
reporting that 
training activity to 
be helpful. The 
overview of 



Overview of PRM 

PRM Application in 
Allegheny County 

Policy/Practice 
Case Scenarios 
KIDS Changes/Design 
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predictive risk modeling was rated the third most helpful with 21% of the survey participants 
indicating that it was helpful. Although the sections on changes to KIDS and those made to 
policy/practice were rated as helpful by fewer call screeners, they were still thought to be 
important components of the training. 

Call screeners were asked what could have been done differently to better prepare them to use 
the Allegheny Family Screening Tool. Over 40% of the screeners offered no comment or said 
that nothing additional was needed. The remaining respondents gave a wide range of open- 
ended responses, including that the tool should have been tested by Intake prior to roll-out, or 
at least call screeners should have been able to provide input into its design. One staff member 
indicated additional training would be beneficial, while another noted a handout explaining the 
information would have been sufficient. 

Other internal stakeholders were also asked what could have been done differently to better 
prepare staff to use the Allegheny Family Screening Tool. One staff member from the Quality 
Assurance, Best Practices and CYF Analytic teams suggested that the concept of the score not 
being about the current allegations "needs to be said often because people forget." The score 
takes into account the history of household members, along with the current allegation. 
Another team member suggested clarification was needed about the differences in the 
definition of "risk." This stakeholder pointed out that the definition of "risk" according to the 
tool (future risk of subsequent allegations of maltreatment or placement into foster care) is 
different from the definition of "risk" of which staff are most familiar (imminent risk of serious 
harm). 

Post-Implementation: Reactions and Process Improvements 

The overall goal of implementing predictive risk modeling in Allegheny County was to improve 
decision-making by making it more data driven and creating a uniform approach to making 
decisions, while increasing the number of those responsible to carry out that activity. Prior to 
implementing PRM, supervisors were responsible for making final call screening decisions. Since 
implementation, all of the call screeners as well as the supervisors are involved in making call 
screening decisions. 

Use of Data 

The follow-up survey conducted of call screeners also asked about the extent to which they 
conduct a more thorough search in the data warehouse, as well as KIDS, this time with a focus 
on reports in which the Family Screening Score was high. A little more than 60% of the call 
screeners who responded to the survey noted they "rarely" or "never" conduct an additional 
search in ClientView or KIDS. Figure 3 illustrates the extent to which call screeners report 
conducting subsequent searches in ClientView when the resulting score is high. Overall, full¬ 
time screeners were more likely to conduct additional searches in ClientView than part-time 
call staff. When asked to explain why additional searches are not done, most call screeners said 
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that they had done the searches in the data warehouse earlier in the process and one call 
screener noted that "the score stands for what is pulled" by the PRM tool. 


Figure 3. Additional Searching Conducted 
When Family Screening Score is High 


50% 

45% 



BClientView OKIDS 


Call Screener Attitudes and Beliefs 


When asked how confident call screeners are in the tool's ability to accurately assess the risk of 
placing a child into out-of-home care or incurring a repeat re-referral of maltreatment, half of 
the call screeners said they were confident. Full-time call screeners were slightly less likely to 
agree than the part-time call screeners. One screener explained that lack of confidence in the 
PRM tool stems from its inability to take families' expected improvement or individual 
circumstances into account, for instance, when families are receiving services that are 
improving their situation. When asked how confident they were in the research that went into 
developing the tool, 11 of the 18 call screeners (61%) reported they were confident in the 
research that went into developing the tool. 

A series of statements were included in the follow up survey which were used to gauge the call 
screeners' understanding of the Family Screening Tool. As shown in Table 3, call screeners 
understand the intention of using the tool in making screening decisions. 


Table 3. Attitude of Call Screeners Toward the Allegheny Family Screening Tool 

Percentage of Call Screeners "Agreeing" or "Strongly Agreeing" with Statement 

1 understand what the score is predicting. 

94% 

1 understand how the score should relate to or inform screening decisions. 

94% 

1 understand the content of the data sources helping to produce the score. 

89% 

1 am adequately prepared to use the tool. 

100% 


Call screeners were also asked how easy or difficult it is to use the Family Screening Tool. Over 
60% of the call screeners who responded to the surveys, regardless of when administered, find 
the Family Screening Tool "easy" or "very easy" to use, as displayed in Figure 4. Many of the 
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respondents appreciate that the resulting score generated by the tool helps to validate their 
decision to screen in or out the General Protective Services referral for investigation. 


Figure 4. Ease of Navigation/Use of 
the Allegheny Family Screening Tool 


70% 69% 



Difficult/Very Difficult Neutral Easy/Very Easy 

□ Initial Survey n Follow-up Survey 


A graphical display, using a thermometer, is used to highlight the value of the score. Call 
screeners were asked in the first survey to indicate how useful the thermometer was to them. 
While many of the call screeners (38%) said they had "no opinion" about the graphic display, 
nearly half (44%) found the thermometer to be "somewhat helpful" or "helpful," explaining 
that it is straightforward and easy to read. An additional 19% said the graphic was "not helpful 
at all" or was only "limitedly helpful," noting the number could be larger and the color scheme 
could be improved. One call screener said that an actual number would suffice while another, 
who self-identified as a visual learner, liked everything about the graphic display. 

Call screeners were also asked to express concerns they had about the screening tool. One of 
the call screeners, as noted earlier, stated that "the tool does not take the human element of 
judgment into account," while another stated that "the score frequently has nothing to do with 
what is actually going on with the situation at hand." Another said that call screeners are able 
to recognize information that needs to be updated, which the tool is unable to do and thus will 
generate a score that inaccurately portrays a family's circumstances. 

Stakeholder Input 

Following implementation of PRM in Allegheny County, stakeholders internal and external to 
DHS continue to have a positive reaction to Allegheny County's implementation of the Family 
Screening Tool. Their hope is that the tool will result in the increased safety of children and 
enable the County to be more proactive and less reactive in its case practice. An internal 
stakeholder noted that the tool should help with decision making, especially for borderline 
cases, such as when it is difficult to determine whether the case should be screened in for an 
investigation or screened out and possibly referred to community services. It should be noted 
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that the goal of the PRM implementation is to use the tool to make decisions regarding every 
GPS referral. 

The family court judge who participated in the post-implementation interviews stated that she 
supports use of the tool. While admittedly she does not have the knowledge needed to 
examine the algorithm used to generate the score, she has confidence in the people who 
presented the material at the community presentation she attended and in what the algorithm 
is meant to achieve. She stated it was clear that the County and its team of experts know what 
they are doing and that inviting stakeholders to the presentation demonstrates that DHS 
intends to be transparent in its implementation of the tool. 

Another external stakeholder explained how her foundation was approached to fund the 
initiative, in part, based on a longstanding relationship with DHS. This stakeholder went on to 
say that DHS is going beyond what is required in terms of keeping the foundation informed and 
those at the foundation welcome the additional information and the County's transparency. 

Both internal and external stakeholders noted during the interviews that there is a need to 
continue to keep community stakeholders informed. For example, one analyst internal to DHS 
thought that the community needs to remain involved and informed as the County moves 
forward with predictive analytics. An external stakeholder noted that the PRM community 
presentation has not evolved in the last two years; the same information is presented in the 
same manner at each meeting. One provider said it should be stressed that the score is just an 
additional piece of information to further assist with decision making—it is not the only factor 
considered. Another provider wanted to know what the "disaster plan" is for the Family 
Screening Tool, e.g., what safeguards are in place to ensure that transparency will continue in 
the future, regardless of who is overseeing the project. 

Process Improvements 

Impact on Practice 

The surveys administered to call screeners two months following implementation and then four 
to five months later gave them the opportunity to comment on whether they thought the tool 
would have an impact on the call screening practice. The follow-up survey also asked screeners 
to explain why or why not it has had an impact, with half (n=9) offering an explanation. While 
the initial survey found half (50%) of the call screeners did not anticipate PRM to have any 
impact, this changed slightly (44%) in the later survey. Some call screeners explained that 
mandatory screen-ins based on a high score would impact practice. Others commented that 
scores should not be mandatory on active cases and decisions to assess a referral are still based 
on the allegations presented by the caller. As displayed in Figure 5, 44% of the call screeners 
overall, when results from both surveys are considered, thought using the tool would 
strengthen the call screen practice. Call screeners stated "consistent decision making will be 
increased." One call screenerfrom the initial survey responded PRM would diminish practice. 
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while two responded as such to the follow-up survey. This may be due to the reported slowness 
of the system which may have become more of an irritant over time. 


Figure 5. Expected Impact on Call Screener Practice 



■ Initial Survey ■ Follow-up Survey 


One of the stated goals of implementing PRM was to create a workplace culture that is more 
data-driven. Call screeners were asked their perspective on whether the use of the predictive 
risk modeling tool was shifting the workplace culture towards that goal. The follow-up survey 
administered to call screeners found that 61% of the screeners, as shown in Figure 6, either 
agreed or strongly 
agreed that the tool 
is creating a data- 
driven culture 
within the 
workplace. When 
this finding is 
considered along 
with the percentage 
of call screeners 
who said the tool 
would not impact 
call screener 
practice (44%), it is 
possible that call 

screeners already thought of the culture at Allegheny DHS as being data-driven. It appears the 
decisions being made by the screening unit prior to PRM implementation were based on good 
screening practices, with the tool now reinforcing those decisions through a systemized use of 
data. 


Figure 6. Use of the PRM Tool is Changing the 
Culture of Our Workplace to Be More Data-Driven 



I Strongly Disagree "Disagree "Agree " Strongly Agree 
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Improved Screening Process 


The survey administered to call screeners two months following implementation of the tool 
found that over 40% of the call screeners use the tool to inform their recommendation on a 
consistent basis, although a little less than a third (31%) reported they rarely use it, if at all. On 
the survey administered after enhancements were made to the tool, a smaller percentage of 
staff said they "always" used it, although a larger percentage "almost always" or "occasionally" 
use the tool to inform their recommendation. When asked, in the follow-up survey, to explain, 
three of the call screeners who did not use the tool to inform their recommendation said their 
recommendation was already formed prior to running the score. One call screener reported 
that he or she 
would discuss the 
score or concerns 
about the score 
with the 
supervisor while 
another said that 
he or she would 
provide the 
information to the 
supervisor to 
make the 
decision. One of 
the call screeners 
stated that "the 
score has nothing 

to do with assignment; active cases are always high or mandatory." Figure 7 illustrates the shift 
in call screeners use of the tool to inform their recommendation. 

Technical Improvements 

Call screeners were also asked in the survey administered after the tool was enhanced how 
often a score was generated that seemed inaccurate given the information they had gathered 
about the family based on the data which are available and/or from what they had collected 
from the reporter. Nearly three-quarters (72%) of the call screeners noted they "occasionally" 
have encountered a score that seems to be inaccurate, while an additional 11% have frequently 
encountered an inaccurate score. 

When asked to explain what they do when the score appears to be inaccurate, nearly half (n = 

9) said they notify a supervisor. Three staff, two full-time and one part-time call screener, 
reported they review KIDS and/or ClientView to inform their decision when they believe the 
score is not right. Another screener relied on the research he or she already completed, instead 


Figure 7. Use of Tool to Inform Recommendation 



Never Almost Occasionally Almost Always 

Never Always 

■ Initial Survey □ Follow-up Survey 
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of relying on the tool to assist with that process. Yet another staff member reported contacting 
the technology staff regarding the inaccurate score. 

Suggestions offered by call screeners during both surveys to improve the Allegheny Family 
Screening Tool were primarily related to technical issues. Several call screeners from the follow¬ 
up survey remarked the system is slow; staff noted it takes a long time for the score to 
generate, with the system sometimes "timing out." One staff member suggested the risk and 
safety boxes should not be locked after the score is viewed so that recommendations could be 
adjusted later in the process. Another screener suggested that the tool should also pull the 
family history with CYF into the narrative field which lists the factors used to produce the score, 
not just the program areas identified using ClientView. 

While these suggestions may improve the functionality of the tool at some time in the future, 
there have been two technical issues reported that specifically affected the performance of the 
tool. A few call screeners noted concerns about the accuracy of the score when clients and 
Master Client Index (MCI) numbers^^ are duplicated within a referral. A similar issue was 
identified with the tool not being able to generate the score when MCI numbers were missing. 
These issues were corrected in November 2016; the follow-up survey identified that most 
screeners (83%) find the score is "mostly" or "always" clearly displayed. 


Master Client Index or MCI numbers are used by DHS staff to identify individuals known to other agencies and 
providers across Allegheny County. 
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Recommendations 


Major changes in policy and practice can be difficult to implement, particularly when the 
agency making the change is pioneering a new technological solution, such as predictive risk 
modeling. Allegheny County has chosen to implement a PRM tool to increase appropriate and 
consistent use of data to drive its decision-making. While the results of this process evaluation 
are encouraging, some recommendations are offered toward informing the implementation 
process moving forward. 

1. Maintain transparent communication with internai and externai stakehoiders. 
Stakeholders overwhelmingly applauded the efforts that DHS has made to be 
transparent and to keep them informed throughout the implementation process. It will 
be important that this transparency continue. 

2. increase user buy-in. 

Less than half of the call screeners currently view predictive modeling as benefiting 
screening practice, though more than 60% agreed that the tool is creating a data-driven 
culture within the workplace. An opportunity exists to increase user buy-in. 

3. Continue to resoive technicai issues as they arise, documenting soiutions. 

As changes and enhancements are made to the tool, they should be documented to 
inform further tool development, increasing the return on the technological investment. 

4. Deveiop benchmarks for impiementing predictive risk modeiing. 

Benchmarks can be developed to foster buy-in and promote use of the tool for decision¬ 
making. Using the results of the process evaluation, Allegheny might consider 
developing benchmarks which would target an increase in the percentage of staff who 
use the tool on an ongoing basis. For example, one measure might challenge call 
screeners to consistently use the tool in their decision-making, e.g., by March 31, 2018 
85% of all call screeners report using the tool always or almost always to inform their 
recommendation to screen in or out a GPS referral for assessment. Benchmarks to keep 
stakeholders informed might also be considered to ensure transparency does in fact 
occur, e.g., issue quarterly newsletters to external stakeholders to keep them informed 
on progress. 
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Appendix A: Initial Survey Results 


Demographics 


Characteristics of Call Screeners Participating in Survey 

Work Status 


Full time 

Part time 

Unknown 

Number of Call Screeners Surveyed by Work Status 

12 

4 

0 

Average Years Worked as a Call Screener by Work Status 

8 

5 

0 

Average Years Worked for Allegheny County by Work Status 

14 

7 

0 


Training 

Did you receive training prior to using the Allegheny Family Screening Tool? 

# 

Yes 

14 

No 

0 

No Answer 

2 

Total 

16 


How well did the training prepare you to use the Tool? 

# 

Not at all 

1 

Limitedly 

1 

No opinion 

3 

Somewhat 

5 

Very well 

6 

Total 

16 


How well did the training increase your understanding about how the Tool 
works? 

# 

Not at all 

1 

Limitedly 

1 

No opinion 

3 

Somewhat 

5 

Very well 

6 

Total 

16 


Which part of the training did you find to be most helpful? 

# 

No answer 

2 

Overview of Predictive Analytics/Predictive Risk Modelling 

3 

Application of Predictive Analytics in Allegheny County 

5 

Policy/Practice 

1 

Case scenarios 

4 

Process changes 

0 

KIDS Changes/Design 

1 

Total 

16 
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Screening Tool Function and Visualization 


How easy/difficult is it for you to navigate or use the PRM tool? 

# 

Very difficult 

0 

Difficult 

0 

Neutral 

5 

Easy 

4 

Very easy 

7 

Total 

16 


How helpful is the "thermometer" visualization? 

# 

Not helpful at all 

2 

Limitedly helpful 

1 

No opinion 

6 

Somewhat helpful 

4 

Very helpful 

3 

Total 

16 


Decision Making 


How frequently do you use the PRM tool to inform your recommendation 
(excluding mandatory referrals)? 

# 

Never 

1 

Almost never 

4 

Occasionally/sometimes 

4 

Almost every time 

4 

Every time 

3 

Total 

16 


How helpful is the PRM tool to inform your recommendation? 

# 

Not at all helpful 

1 

Limitedly helpful 

2 

Neutral/No opinion 

6 

Somewhat helpful 

7 

Very helpful 

0 

Total 

16 


How often do you conduct a search in ClientView after viewing the dynamic 
text ("mad libs")? 

# 

Never 

1 

Almost never 

3 

Occasionally/sometimes 

4 

Almost every time 

4 

Every time 

4 

Total 

16 
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What impact do you think the Allegheny Family Screening Tool will have on 
the call screen practice? 

# 

Strengthen the practice 

7 

Diminish the practice 

1 

No impact on the practice 

8 

Total 

16 
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Appendix B: Follow-up Survey Results 


Demographics 


Characteristics of Call Screeners Participating in Survey 

Work Status 


Full time 

Part time 

Unknown 

Number of Call Screeners Surveyed by Work Status 

11 

6 

1 

Average Years Worked as a Call Screener by Work Status 

9 

6 

1 

Average Years Worked for Allegheny County by Work Status 

15 

9 

3 


Experience and Attitudes with Using the Allegheny Family Screening Tool 


How easy/difficult is it for you to navigate or use the Allegheny Family 
Screening Tool? 

# 

Very Difficult 

0 

Difficult 

1 

Neutral 

6 

Easy 

4 

Very Easy 

7 

Total 

18 


How often have you had a score that seems inaccurate given the family 
history you have available or collected during your review of the call? 

# 

A great deal 

0 

A moderate amount 

2 

Occasionally 

13 

Rarely 

3 

Never 

0 

Total 

18 


Agreement with Statements 

Always 

Mostly 

Sometimes 

Rarely 

Never 

Total 

The score is clearly displayed. 

7 

8 

3 

0 

0 

18 

1 go back and conduct a more thorough 
search in KIDS when the score is high. 

2 

3 

2 

5 

6 

18 

1 go back and conduct a more thorough 
search in ClientView when the score is high. 

3 

1 

3 

6 

5 

18 

How frequently do you use the Tool to 
inform your recommendation (excluding 
mandatory referrals)? 

2 

5 

6 

4 

1 

18 
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Agreement with Statements 

No 

Answer 

Strongly 

Disagree 

Disagree 

Agree 

Strongly 

Agree 

Total 

1 am confident of the Tool's ability to accurately 
assess risk of future placement or re-referral. 

0 

1 

8 

7 

2 

18 

1 understand what the score is predicting. 

0 

0 

1 

9 

8 

18 

The use of this tool is changing the culture of 
our workplace to be more data-driven. 

0 

1 

6 

9 

2 

18 

1 understand how the score should relate to or 
inform screening decisions. 

0 

0 

1 

13 

4 

18 

1 understand the content of the data sources 
helping to produce the score. 

0 

0 

2 

10 

6 

18 

1 am adequately prepared to use the Tool. 

1 

0 

0 

11 

6 

18 

1 am confident in the research that went into 
the development of this tool. 

0 

1 

6 

9 

2 

18 


What impact do you think the Allegheny Family Screening Tool will have on 
the call screen practice? 

# 

Strengthen the practice 

8 

Diminish the practice 

2 

No impact on the practice 

8 

Total 

18 
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